Computational Calculus Series

Part 10: Quadratic functions

Quadratic functions naturally arise when studying second order derivatives, and then by extension second order Taylor series approximations.

There is nothing complicated about a quadratic of a single input, but as we go up in dimension quadratics can become significantly more complex both in terms of the variety of shapes they can take as well as their general formalities.

Here we explain these complexities by discussing general quadratic functions, various ways of thinking about their construction, and the factors which control their shape.

Press the botton 'Toggle code' below to toggle code on and off for entire this presentation.

from IPython.display import display
from IPython.display import HTML
import IPython.core.display as di # Example: di.display_html('<h3>%s:</h3>' % str, raw=True)

# This line will hide code by default when the notebook is exported as HTML
di.display_html('<script>jQuery(function() {if (jQuery("body.notebook_app").length == 0) { jQuery(".input_area").toggle(); jQuery(".prompt").toggle();}});</script>', raw=True)

# This line will add a button to toggle visibility of code blocks, for use with the HTML export version
di.display_html('''<button onclick="jQuery('.input_area').toggle(); jQuery('.prompt').toggle();">Toggle code</button>''', raw=True)

Toggle code

1. Constructing quadratic functions from simple pieces

1.1 Single input quadratic functions

The basic formula for a quadratic of a single input takes the familiar form

\begin{equation} g(w) = a + bw + cw^2 \end{equation}

• here the sign of $c$ controls whether or not the quadratic is convex (points upwards) or concave (points downwards), while its magnitude controls how quickly the quadratic grows

Below we plot two simple quadratics centered at the origin $g(w) = 6w^2$ and $g(w) = -w^2$

# create two quadratic functions
func1 = lambda w: 6*w**2 
func2 = lambda w: -w**2

# use custom plotter to show both functions
title1 = '$g(w)=$3w^2$'; title2 = '$g(w)=$-w^2$';
linlib.plotters.double_2d_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,fontsize = 18,color = 'lime')

In three dimensions - where we have two inputs $w_1$ and $w_2$ - the same quadratic formula can be defined using one of the inputs. For example, using $w_1$ the formula takes the form

\begin{equation} g(w_1,w_2) = a + bw_1^{\,} + cw_1^2 \end{equation}

In three dimensions this looks precisely like $g(w) = a + bw_1^{\,} + cw_1^2$ stretched along the $w_2$ input dimension - forming a half-pipe.

We plot the two and three dimensional versions of this quadratic side by side below.

# plot a single input quadratic in both two and three dimensions
func1 = lambda w: w**2 
func2 = lambda w: w[0]**2

# use custom plotter to show both functions
title1 = '$g(w)=$w^2$'; title2 = '$g(w_1,w_2)=w_1^2$';
linlib.plotters.double_2d3d_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,fontsize = 18,color = 'lime')

• we can define this single input quadratic along any dimension we want

• in general we have $N$ possible inputs $\mathbf{w}=[w_1,\,w_2,\,\ldots,\,w_N]$ we can define it along the $n^{th}$ dimension as $g(\mathbf{w}) = a + bw_n^{\,} + cw_n^2$

1.2 Constructing multi-input quadratics when $N=2$

• with multiple inputs we can form more complex quadratic functions by summing up a number of single input quadratics like the ones discussed above

• for example, with $N=2$ inputs we can form two single input quadratics (one along each input dimension)

\begin{array} \ g_1\left(w_1^{\,},w_2^{\,}\right) = a_1^{\,} + b_1^{\,} w_1^{\,} + c_1^{\,} w_1^2 \\ g_2\left(w_1^{\,},w_2^{\,}\right) = a_2^{\,} + b_2^{\,} w_1^{\,} + c_2^{\,} w_1^2 \\ \end{array}

• adding these together gives us a more complex quadratic $g(w_1,w_2) = g_1(w_1,w_2) + g_2(w_1,w_2)$

• the convexity/concavity along each input dimension is explicitly controlled by its corresponding single input quadratic

• for example, in below we plot the following single input quadratics

\begin{array} \ g_1(w_1,w_2) = w_1^2 \\ g_2(w_1,w_2) = -w_2^2 \\ \end{array}

as well as their sum (notice: this quadratic is convex along $w_1$ and concave along $w_2$)

\begin{equation} g(w_1,w_2) = w_1^2 - w_2^2 \end{equation}

# plot a single input quadratic in both two and three dimensions
func1 = lambda w: w[0]**2 
func2 = lambda w: -w[1]**2

# use custom plotter to show both functions
title1 = '$g_1(w_1,w_2)=w_1^2$'; title2 = '$g_2(w_1,w_2)=-w_2^2$'; title3 = '$g(w_1,w_2)=w_1^2-w_2^2$';
linlib.plotters.triple_3dsum_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,title3=title3,fontsize = 18,color = 'lime')

If instead we choose

$$ \begin{array} \ g_1(w_1,w_2) = w_1^2 \\ g_2(w_1,w_2) = w_2^2 \\ \end{array} $$

and plot them along with their sum

\begin{equation} g(w_1,w_2) = w_1^2 + w_2^2 \end{equation}

we see that the sum is convex along each coordinate axis, since each individual single input quadratic was convex.

# plot a single input quadratic in both two and three dimensions
func1 = lambda w: w[0]**2 
func2 = lambda w: w[1]**2

# use custom plotter to show both functions
title1 = '$g_1(w_1,w_2)=w_1^2$'; title2 = '$g_2(w_1,w_2)=-w_2^2$'; title3 = '$g(w_1,w_2)=w_1^2+w_2^2$';
linlib.plotters.triple_3dsum_plot(func1 = func1, func2 = func2,title1 = title1,title2=title2,title3=title3,fontsize = 18,color = 'lime') 

• below we animate a smooth transition between $g(w_1,w_2) = w_1^2 - w_2^2$ and $g(w_1,w_2) = w_1^2 + w_2^2$

• that is, we gradually increase the coefficient on the $w_2^2$ from $-1$ to $+1$

• you can transition from one to the other by moving the slider back and forth

# animate a smooth transition between g = w_1^2 - w_2^2 and g = w_1^2 + w_2^2
# note: restart kernel if ggplot renderer has already been used (in any of the previous plots) before running - otherwise frames are very low resolutio
linlib.transform_animators.quadratic_3d_flexer.draw_it(num_slides = 100) 

• the simple method detailed above for constructing multi-input quadratics can be used to produce every possible multi-input quadratic - up to orthogonal transformation (e.g., rotation) of the input

• in other words by summing two single-input quadratics, and then possibly transforming the input using an orthogonal matrix, we can generate every possible multi-input quadratic

• to see how this is done first combine terms

\begin{equation} g(w_1,w_2) = \left(a_1 + a_2\right) + \left(b_1 w_1 + b_2 w_2\right) + \left(c_1 w_1^2 + c_2 w_2^2 \right) \end{equation}

• then write more compactly in vector/matrix notation as

\begin{equation} g(\mathbf{w}) = a_{\,}^{\,} + \mathbf{b}_{\,}^T \mathbf{w}_{\,}^{\,} + \mathbf{w}^T \mathbf{C}_{\,}^{\,} \mathbf{w}_{\,}^{\,} \end{equation}

where $\mathbf{w} = \begin{bmatrix} w_1 \\ w_2 \end{bmatrix}$, $\mathbf{C} = \begin{bmatrix} c_{1} \,\,\,\, 0 \\ \,\,0 \,\,\,\, c_{2} \\ \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix}$, and $a = a_1 + a_2$.

• to construct e.g., rotation of coordinates multiply input by orthogonal matrix $\mathbf{V} as ubstituting

\begin{equation} g(\mathbf{Vw}) = a_{\,}^{\,} + \mathbf{b}_{\,}^T \left(\mathbf{Vw}_{\,}^{\,}\right) + \left(\mathbf{Vw}\right)^T \mathbf{C} \left(\mathbf{Vw}^{\,}\right) \end{equation}

• for example, to apply rotation to

\begin{array} \ g_1(w_1,w_2) = w_1^2 \\ g_2(w_1,w_2) = -w_2^2 \\ \end{array}

• write in compact form and use the two-dimensional rotation matrix

\begin{equation} \mathbf{V} = \begin{bmatrix} \text{cos}(\theta) \,\,\,\, -\text{sin}(\theta) \\ \text{sin}(\theta) \,\,\,\,\,\,\,\,\,\, \text{cos}(\theta) \\ \end{bmatrix} \end{equation}

• for each $\theta$ we have a rotated version of the original multi-input quadratic

• we animate this example below over a range of $\theta$ values

# rotate a quadratic 
# note: restart kernel if ggplot renderer has already been used (in any of the previous plots) before running - otherwise frames are very low resolutio
func = lambda w: w[0]**2 - w[1]**2
linlib.transform_animators.quadratic_3d_rotater.draw_it(func = func,num_slides = 100,color = 'lime') 

The general $N$ dimensional case

• this works precisely as described above for $N = 2$

• for further details on this see the associated post